Disk-Based Successive Sampling for Outlier Detection in High Dimensional Data

We propose a sampling based outlier detection method for large high-dimensional data. Our method consists of two phases. In the first phase, we combine a “successive sampling” strategy with a simple randomized partitioning technique to generate a candidate set of outliers. This phase requires one full data scan and the running time has linear complexity with respect to the size and dimensionality of the data set. An additional data scan, which constitutes the second phase, extracts the actual outliers from the candidate set. The running time for this phase has complexity where and are the size of the candidate set and the data set respectively. A major strength of the proposed approach is that no partitioning of the dimensions is required thus making it particularly suitable for high dimension data. Furthermore our method can handle both continuous and categorical attributes. We also present a detailed experimental evaluation of our proposed method on real and synethetic data sets. General Terms Outlier Detection, Sampling, High Dimension, Randomization